Description

Add a page to the streamlit dashboard available via dlt pipeline NAME show. It uses the recent Pipeline.dataset interface to pass a dataframe to pygwalker.

Goals

When first loading a pipeline, I want to know the properties of the data (null, distribution, outliers). It's reducing the friction of having to create temporary notebooks and install ipykernel/jupyter dependencies

Implementation

• pygwalker is an open-source Python library that produces an interactive widget to explore dataframes (works in Jupyter, VSCode, Streamlit, Marimo) based on Vega.

• Integration surface area is small; only assumes that you can pull the data locally via the Pipeline.dataset feature.

  • Deeper integration could use Pipeline.dataset to query data or pass an sqlalchemy connection to pygwalker to handle data loading.
  • pygwalker can use duckdb for efficient processing of data

TODO

• add useful message for optional/missing dependencies